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CRITICAL  THINKING  TRAINING  FOR  ARMY  OFFICERS  VOLUME  ONE;  OVERVIEW 
OE  THE  RESEARCH  PROGRAM 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Advanced  training  in  critical  thinking  (CT)  is  needed  for  adult  populations  in  many  fields 
of  work.  Not  surprisingly,  the  United  States  military  is  at  the  forefront  of  efforts  to  promote  and 
improve  thinking  skills.  Military  leadership  demands  the  application  of  high  quality  CT  for 
effective  battle  command.  However,  a  theory  based,  comprehensive,  and  widely  available 
program  of  training  is  needed.  Moreover,  the  scientific  literature  on  critical  thinking  is  highly 
fragmented  and  no  dominant  or  consensus  model  of  the  construct  has  been  developed.  To 
address  the  issues  surrounding  the  concept  of  CT  and  its  development,  the  research  summarized 
in  this  report  focused  on  two  central  objectives.  The  first  objective  was  to  develop  a  predictive 
model  of  CT  that  enhanced  our  understanding  of  the  construct  and  could  be  used  to  guide  future 
research.  The  second  objective  of  the  research  was  to  develop  a  distance-based  training  system, 
designed  to  improve  the  CT  skills  of  Army  officers. 

This  report  is  the  first  in  a  series  of  three  volumes  that  describe  a  multi-year  research 
program  to  develop  and  assess  a  web-based  system  for  training  critical  thinking  for  Army 
officers.  This  volume  gives  a  general  overview  of  the  research  program.  The  second  volume 
(Eischer,  Spiker,  &  Riedel,  2008a)  describes  in  more  detail  the  development  and  validation  of  a 
model  of  critical  thinking  and  the  selection  of  critical  thinking  skills  to  train.  The  third  volume 
(Eischer,  Spiker,  &  Riedel,  2008b)  documents  lessons  learned  in  teaching  critical  thinking 
appearing  in  the  literature  and  lessons  learned  by  Army  instructors  and  curriculum  developers  of 
critical  thinking  training.  It  also  describes  the  requirements  and  course  curriculum  for  the 
prototype  training  system. 

Procedure: 

A  predictive  model  of  CT  was  developed  and  two  research  projects  were  conducted  to 
validate  the  model.  The  first  project  was  an  experimental  test  of  a  number  of  the  hypotheses  of 
the  model.  The  second  validation  project  examined  whether  the  model  could  be  applied  to  Army 
battle  command  and  identified  eight  CT  skills  that  are  both  important  and  problematic  to  Army 
battle  command.  The  model  and  the  eight  skills  were  then  used  to  guide  development  of  the 
training  curriculum.  A  web-based,  prototype  training  program  was  then  developed  that  targeted 
development  of  two  of  the  eight  CT  skills. 

To  develop  the  training  curriculum,  teaching  methods  first  were  identified  that  were 
likely  to  be  most  effective  at  developing  CT  in  Army  officers.  Two  investigative  strategies  were 
adopted  to  determine  the  best  methods.  Eirst,  lessons  learned  from  the  research  literature  on 
training  CT  and  distance-based  learning  were  identified.  Second,  Army  instructors  and 
instructional  developers  who  were  involved  in  teaching  CT  were  surveyed  to  identify  their 
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successes  and  the  problems  they  eurrently  faee.  The  results  from  the  two  researeh  strategies  were 
then  used  to  establish  funetional  requirements  for  web-based  CT  instruetional  material. 


A  web-based  instruetional  system  targeting  CT,  Computerized  Training  in  Critical 
Thinking  or  (CT)  ,  was  then  developed  using  the  functional  requirements  to  guide  its  design.  A 
set  of  pedagogieal  prineiples  derived  from  the  literature  review,  survey  results,  funetional 
requirements,  and  praetieal  eonsiderations  served  as  the  basis  of  the  (CT)^  design. 

To  evaluate  the  effeetiveness,  usability,  and  student  aeeeptanee  of  (CT)^,  an  empirieal 
investigation  was  eondueted  in  whieh  partieipants  from  the  85th  Reserve  Training  Division 
worked  through  parts  of  the  training.  The  eentral  objeetive  of  the  evaluation  was  to  determine 
whether  the  CT  training  system  effectively  inereased  measurable  indicators  of  CT  compared  to 
two  other  training  eonditions.  The  researeh  also  assessed  partieipating  students’  attitudes  and 
subjeetive  evaluations  of  (CT)  as  indieators  of  aeoeptanee  and  usability. 

Findings: 

The  model  that  was  developed  ineorporates  many  ideas  about  CT  offered  by  leading 
thinkers  in  philosophy  and  education.  It  includes  many  of  the  CT  skills  and  predisposing 
attitudes  discussed  in  the  CT  literature.  It  also  speeifies  the  relationships  among  a  variety  of 
variables  that  previous  researehers  have  discussed,  sueh  as  the  influenee  of  experienee  and 
knowledge,  and  the  relationship  of  CT  to  eognitive  tasks  (e.g.,  judgment  and  problem  solving). 
The  model,  however,  goes  beyond  the  largely  rational/analytic  work  eondueted  to  date  by 
providing  a  framework  in  which  CT  can  be  empirieally  investigated  as  a  cognitive  proeess. 

The  validation  researeh  tested  several  predietions  of  the  model.  Many  of  the  findings 
supported  the  premises  of  the  model,  but  some  indieated  the  need  for  greater  detail  of  specifie 
model  elements.  The  results  of  the  seeond  validation  effort,  which  tested  whether  the  model 
eould  be  applied  to  Army  battle  eommand,  indicated  that  the  CT  model  largely  eaptures  the 
skills,  situational  conditions,  and  predisposing  faetors  signifieant  to  Army  battle  eommand.  The 
results  of  this  researeh  were  used  to  identify  a  set  of  eight  CT  skills  which  are  important  and 
problematie  to  Army  battle  eommand. 

Based,  in  part,  on  the  model  of  CT  that  was  developed,  an  examination  of  the  literature 
on  training  eritieal  thinking  and  on  a  survey  of  Army  instructors  and  curriculum  developers, 
requirements  were  developed  for  the  training  system.  The  foeus  of  (CT)  is  to  improve  key 
eritieal  thinking  skills  that  support  Army  battle  eommand.  CT  involves  a  deliberate,  systematie 
awareness  of  the  proeess  and  produets  of  one’s  own  thinking.  The  training  program  targets 
awareness  of  one’s  own  thinking  and  eommon  errors  that  people  make  when  they  fail  to  apply 
appropriate  CT  skills.  These  errors  inelude  overlooking  important  details,  misinterpreting 
information,  and  making  ineorreet  assumptions  -  all  of  whieh  ean  lead  to  poor  deeision-making. 
The  training  program  highlights  awareness  of  these  errors  and  teaehes  speeific  teehniques  that 
ean  help  people  overeome  them.  It  presents  the  student  with  real-world  situations  and  asks  them 
to  eomplete  numerous  thinking  exercises  that  require  the  praetiee  and  applieation  of  CT  skills  in 
a  variety  of  realistie  settings. 
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The  results  of  the  evaluation  with  the  85**^  Reserve  Training  Division  indieated  that 
military  students  find  (CT)^  very  aceeptable;  however,  the  findings  are  based  on  a  relatively 
small  sample.  The  material  was  uniformly  positively  rated.  Despite  the  extensive  time 
commitment  the  training  program  requires,  users  found  the  training  interesting  and  well  worth 
their  time.  (CT)  also  appears  to  be  generally  effective  at  encouraging  critical  thinking  with 
regard  to  messages  Army  personnel  must  evaluate.  The  research  showed  that  the  web-based 
training  enhanced  memory  for  messages,  possibly  because  it  encourages  greater  depth  of 
processing.  (CT)^  also  seems  to  inhibit  the  production  of  (potentially  incorrect)  inferences  that  go 
well  beyond  what  is  explicitly  given  in  the  message.  Participants  who  took  the  (CT)  training 
made  significantly  fewer  unjustified  inferences  than  participants  assigned  to  the  other  two 
training  conditions.  Examination  of  the  responses  reveals  that  (CT)^  participants  did  make 
inferences;  however,  they  justified  them  by  pointing  out  explicit  information  given  in  the 
message  that  supported  their  inferences.  Therefore,  (CT)  appears  to  encourage  discrimination  of 
what  is  “known”  or  “given”  from  what  might  be  added  (i.e.,  inferred)  by  the  perceiver. 

Utilization  and  Dissemination  of  Findings: 

The  model  of  CT  generated  a  number  of  predictions  about  critical  thinking  that  had  not 
been  previously  empirically  tested.  The  model  was  sufficiently  specified  to  permit  falsification 
of  many  of  its  assertions.  As  a  result,  current  knowledge  of  CT  has  been  significantly  extended. 
Although  the  results  of  the  validation  research  were  mixed  in  their  support  of  the  model,  it  is 
clear  that  the  model  has  passed  an  important  scientific  criterion  in  that  it  has  generated  testable 
hypotheses  that  have  produced  empirical  findings. 

The  results  of  this  research  program  also  have  practical  implications  for  the  design  of 
information  systems  and  for  educational  purposes  that  seek  to  improve  the  application  of  CT 
skills.  The  CT  skills  identified  in  this  research  as  important  and  problematic  to  Army  battle 
command  may  be  utilized  for  training  and  assessment  purposes  and  to  increase  Soldiers’  self- 
awareness  of  their  thinking.  Training  concepts  derived  in  this  research  and  discussed  in  Fischer, 
et  al.  (2008b)  have  also  been  adopted  at  the  Army  Command  and  General  Staff  College  to  teach 
the  key  CT  skills  identified  by  the  present  research. 

This  report  provides  an  overview  of  an  extended  program  of  research  dealing  with  the 
training  of  CT  for  Army  officers.  It  describes  the  systematic  approach  that  was  used  to  develop  a 
computer  based,  on-line  educational  program  designed  to  promote  the  CT  of  Army  officers  and 
provides  documentation  of  an  empirical  evaluation  of  the  (CT)  .  Researchers,  teachers,  and 
curriculum  developers  can  use  this  report  as  a  stepping-stone  to  the  development  of  future  CT 
educational  programs,  or  to  better  understand  (CT)  for  their  own  use. 
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CRITICAL  THINKING  TRAINING  FOR  ARMY  OFFICERS 
VOLUME  ONE:  OVERVIEW  OF  RESEARCH  PROGRAM 

Introduction 

Background 

Interest  in  promoting  eritical  thinking  (CT)  skills  has  inereased  over  the  past  20  years  in  a 
variety  of  diverse  settings  such  as  public  education,  military  leadership,  nursing,  technical 
vocations,  and  corporate  business  (e.g.  Onwuegbuzie,  Schwartz,  &  Rice,  2000).  CT  has  also 
become  a  recognized  construct  in  philosophy,  education,  and,  to  a  lesser  degree,  psychology. 

Not  surprisingly,  the  United  States  military  is  at  the  forefront  of  the  effort  to  promote  and 
improve  thinking  skills  such  as  critical  thinking  skills.  Military  decision-making  demands  the 
application  of  high  quality  CT  for  effective  battle  command,  where  battle  command  applies  “to 
the  leadership  element  of  combat  power... Commanders  visualize  the  operation,  describe  it  in 
terms  of  intent  and  guidance,  and  direct  the  actions  of  subordinates  within  their  intent.  They 
directly  influence  operations  by  personal  presence...”  (Department  of  the  Army,  2001,  Section 
5-1).  Army  leaders  often  find  themselves  in  situations  that  bear  little  resemblance  to  conflict 
situations  they  have  previously  experienced  or  studied.  In  recent  years,  they  have  been  expected 
to  serve  peacekeeping  roles,  for  example,  in  which  their  job  is  to  control  conflict  among  two  or 
more  opposing  and  hostile  groups  within  a  foreign  country.  In  such  situations,  learned  rules  of 
engagement  and  historical  principles  of  warfare  often  do  not  apply.  There  is  no  single  enemy  and 
battle  lines  may  not  exist.  In  such  situations,  novel  solutions  that  are  the  product  of  CT  are  likely 
to  be  critical  to  success.  Hence,  Soldiers  cannot  simply  base  their  battle  plans  on  well-learned 
battlefield  patterns;  they  must  reason  through  and  integrate  an  enormous  amount  of  information. 
Lessons  learned  from  Operation  Iraqi  Freedom,  for  example,  are  accumulated  and  distilled  into 
new  doctrine  (e.g.,  the  new  counterinsurgency  field  manual,  FM  3-24).  That  doctrine  is 
incorporated  into  training  and  education,  but  the  problem  persists.  It  is  not  possible  to  anticipate 
all  possible  situations  which  will  arise,  and  the  leaders  will  need  to  draw  on  their  thinking  skills 
to  understand  and  develop  responses  in  unfamiliar  situations. 

Extensive  training  time  is  devoted  to  the  teaching  of  content  knowledge  in  the  military. 
Many  hours  are  spent  examining  and  evaluating  historical  battles  to  develop  a  deep 
understanding  of  the  relationships  among  the  principal  factors  affecting  the  outcomes  of  warfare. 
In  contrast,  relatively  little  training  time  has  been  spent  on  improving  the  process  of  thinking  and 
decision-making.  It  is  not  that  the  Army  educational  system  has  neglected  to  provide  instruction 
in  CT  or  reasoning.  However,  fewer  resources  have  been  devoted  to  the  training  of  thinking 
processes  than  to  other  important  skills  (Riedel,  Morath,  &  McGonigle,  2001).  Moreover,  the 
prescriptive  and  procedural  nature  of  the  doctrinal  methods  may  actually  discourage  the 
application  or  development  of  thinking  skills,  inhibiting  the  creation  of  novel  solutions  that 
might  be  the  result  of  CT  (Fallesen,  Michel,  Fussier,  &  Pounds,  1996).  In  short,  the  education  of 
military  commanders  seems  to  be  a  prime  opportunity  for  training  designed  to  foster  thinking 
skills,  and  that  the  domain  of  battle  command  is  one  in  which  CT  is  crucial. 
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The  Research  Program 


The  central  purpose  of  the  research  program  documented  in  this  report  was  to  develop 
training  in  critical  thinking  for  Army  personnel.  This  report  describes  the  systematic  approach  that 
was  used  to  develop  a  computer  based,  on-line  educational  program  designed  to  promote  CT  in 
Army  officers.  Two  companion  volumes  describe  the  research  program  in  more  detail.  This 
volume  presents  a  summary  of  the  series  of  research  efforts  that  culminated  with  the  development 
and  evaluation  of  the  CT  training  for  Army  personnel.  First,  a  theoretical  model  of  critical  thinking 
was  developed  and  validated.  This  model  subsequently  guided  development  of  the  training  system. 
Then  the  role  of  CT  in  Army  battle  command  was  investigated.  Research  was  conducted  to 
determine  the  degree  to  which  the  CT  model  was  applicable  to  the  kinds  of  problems  faced  by 
Army  leaders.  The  research  demonstrated  that  a  number  of  CT  skills  are  both  important  and 
problematic  to  battle  command.  Eight  CT  skills,  in  particular,  were  identified  as  high  impact 
skills  for  battle  command  and  selected  as  the  target  skills  for  the  web  based  training.  Next, 
lessons  learned  from  past  research  and  from  a  survey  of  Army  instructors  and  instructional 
designers  were  analyzed  to  help  define  the  requirements  for  the  CT  training  system. 
Computerized  Training  in  Critical  Thinking  (CT)^.  Finally,  (CT)^  was  designed  implemented, 
and  evaluated. 

Volume  Two  of  this  series  presents,  in  more  detail,  the  model  and  theoretical  basis  for  the 
prototype  training  system  that  was  developed  for  two  of  the  eight  skills  and  the  experiments  that 
were  conducted  to  validate  the  model.  Volume  Three  describes  the  training  that  was  developed, 
including  the  functional  requirements,  pedagogical  principles,  course  content,  and  evaluation  of  the 
training.  A  fourth  report  describes  an  expanded  training  system  that  was  developed  to  train  all  eight 
critical  thinking  skills  (Fischer,  Spiker,  Harris,  McPeters,  &  Riedel,  2008). 


A  Model  of  CT 

Development  of  training  that  effectively  and  efficiently  improve  critical  thinking  skills 
important  for  proficient  battle  command  starts  with  the  identification  of  the  skills  required.  An 
empirically  grounded  model  is  essential  to  any  effort  seeking  a  meaningful  improvement  in  CT 
skills  because  it  would  serve  to  identify  those  skills.  In  addition,  before  effective  training  can  be 
developed,  training  objectives  must  be  specified,  and  those  objectives  should  be  derived  from  an 
empirically  tested  model  of  CT.  However,  no  such  model  existed  prior  to  this  research. 

A  model  of  CT  was  needed  that  was  sufficiently  specific  so  that  it  could  be  subjected  to 
empirical  testing.  The  model  could  then  be  used  to  organize  and  test  many  of  the  hypotheses 
appearing  in  the  literature.  Such  a  model  would  also  point  to  future  areas  of  research. 

The  model  developed  in  the  course  of  this  research  program  (see  Figure  1)  is  based  on  a 
review  of  the  literature  on  critical  thinking  (Fischer,  et  ah,  2008b)  and  incorporates  many  ideas 
about  CT  offered  by  leading  thinkers  in  philosophy  and  education.  It  embodies  many  of  the  CT 
skills  and  predisposing  attitudes  discussed  in  the  CT  literature.  It  also  specifies  the  relationships 
among  a  variety  of  variables  that  previous  researchers  have  discussed,  such  as  the  influence  of 
experience  and  knowledge,  and  the  relationship  of  CT  to  cognitive  tasks  (e.g.,  judgment  and 
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problem  solving).  However,  the  model  goes  beyond  the  largely  rational/analytic  work  eonducted 
to  date  by  providing  a  framework  in  which  CT  can  be  empirically  investigated  as  a  cognitive 
process. 


Generally  stated,  the  model  posits  that  CT  is  a  cognitive  process  that  intervenes  between 
a  set  of  initiating  situational  conditions  and  the  observable  performance  of  one  or  more  tasks.  It 
is  purposeful  and  deliberate  cognitive  processing  that  includes  checks  on  the  process  and 
products  of  thinking  and  serves  other  higher-level  tasks  such  as  decision-making.  CT  involves 
the  application  of  least  one  of  a  particular  set  of  cognitive  skills  that  demand  the  use  of  meta- 
cognitive  and/or  recursive  control,  consciously  controlled  logic,  or  thorough  examination  of  a 
problem.  The  execution  of  CT  skills  is  powered  by  an  effortful,  yet  flexible,  computational 
process  that  is  capable  of  controlled  meta-cognitive  and  recursive  monitoring  of  thinking.  The 
processing  engine  is  distinguished  from  the  quick,  yet  powerful,  recognition-based  processing 
(Klein,  1999)  that  depends  on  association  strengths  and  that  is  subject  to  well-documented 
accessibility  errors  (Kahneman,  2003).  The  application  of  CT  skills  has  accompanying  affective 
consequences,  which  may  be  measured  using  standard  physiological  preparations  or 
psychometric  instruments. 
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The  model  places  CT  in  the  context  of  cognitive  processes  posited  by  the  most  current 
models  of  reasoning  and  judgment  (Kahneman,  2003).  The  model  incorporates  six  major  factors 
involved  in  CT  including,  (1)  situational  conditions,  (2)  meta-tasks,  (3)  CT  skills,  (4) 
predisposing  individual  differences,  (5)  moderating  variables,  and  (6)  negative  experiential 
consequences.  Figure  1  shows  the  components  of  the  model.  Volume  Two  of  this  series  gives  a 
more  detailed  description  of  the  model  and  the  literature  review  that  informed  its  development. 

CT  is  considered  in  this  model  to  occur  within  a  relatively  brief  time  frame  (in  the  range 
of  5  to  30  minutes)  in  which  certain  essential  processing  functions  are  executed,  depending  on 
the  desired  goal.  One  can,  however,  string  together  a  series  of  CT  episodes,  held  together  via 
meta-cognitive  monitoring  processes,  which  would  define  a  much  longer  CT  period  that  might 
encompass  a  training  exercise,  a  class,  or  some  important  real-world  event  (e.g.,  an  entire 
intelligence  briefing).  If  a  given  individual  shows  a  tendency  to  initiate  CT  skills  on  a  frequent 
basis,  he  or  she  might  be  labeled  as  a  “critical  thinker.”  For  a  variety  of  reasons,  individuals  may 
vary  in  the  quality  of  their  critical  thinking,  and  the  quality  of  the  measurable  products  of  the 
application  of  CT  skills.  In  this  model,  critical  thinking  skills  are  considered  both  measurable  and 
modifiable  via  experience.  Therefore,  it  should  be  possible  to  develop  and  evaluate  a  robust 
training  program  to  promote  CT  skills.  Furthermore,  if  one  can  identify  particular  skills  or 
classes  of  skills  that  are  more  commonly  associated  with  high-performance  outcomes,  then  a 
major  effort  to  train  those  skills  should  yield  high-payoffs  for  participating  organizations. 

In  summary,  our  working  definition  of  CT  is:  Critical  thinking  is  a  time-limited  mode  of 
controlled,  deliberate,  processing  that  is  purposeful,  stimulus -driven,  and  context-bound. 

Integral  to  CT  are  checks  on  the  process  and  products  of  thinking,  which  make  it  a  fundamen¬ 
tally  meta-cognitive  process.  Its  function  is  to  serve  other  cognitive  tasks  such  as  decision¬ 
making  and  problem  solving. 
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Experimental  Validation  of  the  CT  Model 


This  section  gives  an  overview  of  the  method  and  results  of  an  experiment  that  examined 
several  predictions  of  the  model.  A  detailed  description  of  the  experiment  is  contained  in 
Volume  Two  of  this  series. 


Predictions  of  the  Model 

An  experiment  was  conducted  to  examine  the  validity  of  several  of  the  model’s 
predictions.  The  first  set  of  predictions  investigated  by  the  experiment  dealt  with  the  role  of 
substantive  information  in  critical  thinking.  The  model  posits  that  the  situation  must  include 
substantive  information,  which  is  the  material  to  which  the  CT  skills  are  applied.  The  model  also 
asserts  that  substantive  information  will  increase  the  tendency  to  employ  CT  skills  compared  to 
less  substantive  information.  The  model  further  states  that  the  use  of  CT  skills  is  more  likely  if 
the  information  presented  by  the  situation  is  conflicting,  disordered,  uncertain,  complex  or 
requires  extensive  logical  reasoning.  To  test  these  predictions,  the  amount  of  content  in  the 
stimulus  material,  as  defined  by  the  number  of  unique  propositions,  was  varied.  Additionally, 
one  condition  was  examined  in  which  inconsistent  (i.e.,  contradictory)  information  was 
incorporated  into  the  stimulus  material. 

A  second  prediction  that  was  tested  concerned  the  purposeful  nature  of  CT.  Specifically, 
the  model  states  that  one  engages  in  CT  skills  only  when  the  situational  context  requires  at  least 
one  of  four  meta-tasks,  i.e.,  when  one’s  task  is  to  understand  some  material,  solve  a  problem, 
make  a  decision,  or  make  an  evaluative  judgment.  To  test  this  hypothesis,  participants  were 
asked  to  perform  three  tasks  (judgment,  understanding,  and  identification  of  the  general  topic)  on 
stimulus  material  that  varied  in  substance. 

A  third  prediction  tested  by  the  experiment  involved  the  effect  of  predisposing  individual 
difference  factors  on  the  application  of  CT  skills.  The  model  states  that  differences  exist  among 
individuals  in  their  tendency  to  use  CT.  To  evaluate  the  influence  of  predisposing  factors, 
participants  were  asked  to  complete  an  assessment  instrument,  the  Need  for  Cognition  Scale 
(NFC)  (Cacioppo,  Petty,  &  Kao,  1984),  which  was  used  as  a  measure  of  the  tendency  to  engage 
in  thought. 

A  fourth  prediction  of  the  model  tested  in  the  experiment  involved  the  effects  of 
moderating  variables  on  CT.  Specifically,  the  model  predicts  that  expertise  should  affect  the 
quality  of  CT,  i.e.,  how  well  someone  executes  a  CT  skill.  However,  it  should  not  affect  whether 
one  attempts  to  perform  that  CT  skill.  To  test  this  prediction,  two  groups  of  participants  were 
recruited  who  varied  in  amount  of  training  in  psychological  research,  namely  undergraduate  and 
graduate  students  in  psychology.  Here,  amount  of  training  was  used  as  an  operational  measure  of 
expertise. 

Finally,  the  experiment  tested  the  model  prediction  that  CT  elicits  negative  affect  because 
it  requires  effort.  The  model  predicts  that  the  application  of  CT  skills  should  be  associated  with  a 
corresponding  increase  in  negative  affect  and  in  effort.  This  prediction  was  tested  by  measuring 
affect  and  effort  immediately  following  each  trial  of  the  experiment. 
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Method 


Twenty-six  participants  (5  males  and  21  females),  ranging  in  age  from  20  to  51  years  of 
age,  took  part  in  the  experiment.  All  participants  had  taken  at  least  one  course  in  experimental 
psychology.  Eleven  were  undergraduate  students  and  15  were  graduate  students.  All  participants 
received  a  test  booklet  containing  nine  problems.  Each  problem  presented  the  participants  with  a 
specific  task  to  apply  to  a  short  paragraph  that  described  a  research  experiment  in  the  area  of 
experimental  psychology.  Over  the  nine  problems,  participants  were  asked  to  perform  three 
different  tasks.  The  task  instructions  asked  participants  to  either  1)  understand,  2)  make  a 
judgment  about,  or  3)  simply  identify  the  general  topic  of  the  material  presented.  The  substantive 
content  of  the  paragraphs  describing  the  nine  research  studies  was  also  varied.  Three  different 
types  of  substantive  content  were  presented.  The  first  type  contained  very  little  substantive 
information,  as  measured  by  number  of  propositions  presented.  The  second  type  was  more 
substantive  than  the  first,  i.e.,  presented  a  greater  number  of  propositions.  The  third  type  was  as 
substantive  as  the  second,  but  also  included  several  propositions  that  were  inconsistent,  i.e., 
contradictory,  with  one  another. 

Participant  measures  included  the  Need  for  Cognition  (NEC)  scale,  self-reported  mental 
effort  and  affect  ratings  for  each  condition,  response  time  for  each  trial,  and  indicators  of  the 
application  of  CT  skills  derived  from  the  verbal  protocols.  Each  of  these  measures  is  relevant  to 
one  or  more  hypotheses  generated  by  the  CT  model. 

Results  of  the  Validation  Experiment 

The  statistical  analyses  for  this  experiment  are  reported  in  Volume  Two  of  this  series.  This 
section  reports  a  summary  of  those  findings. 

Effects  of  Substance  of  Material  on  CT 

As  previously  discussed,  the  model  asserts  that  substantive  information  will  increase  the 
tendency  to  employ  CT  skills  compared  to  less  substantive  information.  A  second  prediction 
concerning  the  substance  of  stimulus  material  is  that  the  use  of  CT  skills  is  more  likely  if  the 
information  presented  is  conflicting,  disordered,  uncertain,  and  complex  or  requires  extensive 
logical  reasoning.  Examining  all  four  measures,  the  results  of  the  statistical  analyses  do  not 
support  the  prediction  that  high  substance  material  increases  the  tendency  to  apply  CT  skills. 
Under  some  task  conditions,  low  substance  material  can  actually  generate  more  CT  than  high 
substance  material.  Specifically,  when  asked  to  make  a  judgment  about  low  substance  material, 
more  questions  of  belief  were  asked  than  when  asked  to  make  a  judgment  about  high  substance 
material. 

The  results,  however,  do  support  the  prediction  that  the  application  of  CT  skills  is  more 
likely  if  the  available  information  is  degraded  in  some  way,  i.e.,  is  conflicting,  disordered, 
uncertain,  etc.  A  general  pattern  emerges  that  inconsistent  material  tends  to  take  longer  to 
process  and  produces  more  questions  of  belief  and  checks  on  thinking. 

These  results  have  practical  implications  for  the  design  of  information  systems  as  well  as 
for  training  purposes  that  seek  to  increase  the  tendency  to  apply  CT  skills.  Designers  should  be 
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aware  that  people  may  not  question  or  check  highly  substantive  material  any  more  than  low 
substantive  material.  If  CT  is  desired,  inconsistent  content  might  be  highlighted  by  information 
systems.  Similarly,  if  training  systems  seek  to  encourage  CT,  one  strategy  would  be  to  sensitize 
students  to  inconsistent  material. 

Effects  of  Task  on  CT 

According  to  the  model,  CT  skills  should  only  be  engaged  when  the  situational  context 
includes  at  least  one  of  the  four  CT  meta-tasks.  The  response  time  and  effort  ratings  supported 
the  predictions  concerning  task  requirements.  The  prediction  was  also  supported  in  that  both  the 
judgment  and  understanding  tasks  generated  more  questions  of  belief  than  did  the  identification 
task.  However,  the  understanding  task  failed  to  generate  more  checks  on  thinking  than  the 
identification  task.  Therefore,  the  prediction  that  understanding  is  a  meta-task  that  elicits  CT  was 
not  supported  by  the  checks  on  thinking  data.  Moreover,  it  appears  that  understanding  and 
judgment  may  not  elicit  CT  equally.  These  results  suggest  that  refinement  of  the  model  may  be 
needed  with  regard  to  task.  While  the  results  generally  support  the  idea  that  understanding  and 
judgment  encourage  the  application  of  CT  skills,  the  results  also  suggest  that  judgment  thus  can 
be  more  effective  at  doing  so. 

Effects  of  Predisposing  Factors  on  CT 

The  model  states  that  differences  exist  among  individuals  in  their  tendency  to  use  CT. 
One  would  expect  a  positive  relationship  between  independent  measures  of  predisposition  and 
indicators  that  CT  has  been  used.  The  results  of  this  experiment  failed  to  support  the  notion  that 
predisposing  individual  difference  factors  affect  the  tendency  to  engage  in  CT  skills.  None  of  the 
correlations  of  the  NFC  scale  with  the  response  measures — response  time,  effort,  affect, 
questions  of  belief,  or  checks  on  thinking  -  achieved  statistical  significance.  While  surprising, 
this  negative  result  may  be  the  consequence  of  a  homogeneous  subject  population — college 
students — ^with  a  high  overall  tendency  to  engage  in  CT.  Or  the  NFC  may  not  be  a  good  measure 
of  tendency  to  use  CT. 

Effects  of  Moderating  Variables  on  CT 

According  to  the  model,  expertise  and  experience  should  affect  the  quality  of  CT  but  not 
the  likelihood  that  one  will  engage  in  CT.  The  model’s  prediction  concerning  experience  was 
supported  in  the  response  time  and  effort  data.  No  differences  were  observed  between  graduates 
and  undergraduates  in  these  two  measures,  nor  was  there  any  difference  in  reported  affect 
between  the  two  groups  of  participants.  However,  the  more  decisive  indicators  of  CT  (i.e., 
measures  of  questions  of  belief  and  checks  on  thinking)  did  support  the  prediction.  Graduate 
students  asked  more  questions  and  checked  their  thinking  more  than  undergraduates,  which 
suggests  they  applied  their  CT  skills  more.  Therefore,  it  appears  that  experience  does  affect  the 
application  of  CT  skills.  However,  this  conclusion  is  weakened  by  the  possibility  that,  on 
average,  graduate  students  are  more  intelligent  than  undergraduates  and  intelligence  is  related  to 
the  likelihood  that  one  will  engage  in  CT. 
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Effects  of  CT  on  Negative  Affect 

According  to  the  model,  the  application  of  CT  skills  should  be  assoeiated  with  a 
corresponding  inerease  in  negative  affect.  Those  conditions  hypothesized  to  elicit  CT  such  as  the 
tasks  of  understanding  and  judgment  and  high  substance  and  ineonsistent  material  should  be 
rated  as  less  enjoyable  than  the  other  eonditions  of  the  experiment.  The  findings  of  the  present 
experiment  were  mixed  in  their  support  of  this  hypothesis.  Participants  reported  that  they 
enjoyed  proeessing  inconsistent  summaries  less  than  the  other  summaries,  whieh  is  congruent 
with  the  model’s  prediction.  However,  the  equivalent  ratings  for  the  low  and  high  substanee 
material  failed  to  support  the  prediction.  In  summary,  negative  affect  was  not  directly  related  to 
CT  in  this  experiment.  Instead,  it  appears  in  this  ease  that  other  factors  determined  the  level  of 
enjoyment  experieneed  by  partieipants. 

Conclusions  of  Model  Validation  Experiment 

The  model  of  CT  generated  a  number  of  predictions  that  had  not  been  previously 
empirically  tested.  The  model  was  sufficiently  speeified  to  permit  falsification  of  many  of  its 
predictions,  a  capability  which  analytic  accounts  of  CT  had  not  provided.  As  a  result,  a  clearer 
pieture  of  the  effects  of  task  and  stimulus  substance  on  CT  has  emerged. 

Although  the  results  of  the  validation  experiment  were  mixed  in  their  support  of  the 
model,  it  is  clear  that  the  model  has  passed  an  important  scientifie  criterion:  it  has  generated 
testable  hypotheses  that  have  produeed  empirical  findings.  This  research  provides  one  test  of  the 
model’s  predictions;  there  are  many  more  hypotheses  to  be  tested  and  the  findings  of  the  present 
research  should  be  examined  further,  replicated,  and  extended.  Some  of  the  findings  point  to 
plaees  in  the  model  that  require  greater  speeification  or  modifieation.  Other  findings  are 
consistent  with  the  model’s  predietions.  Future  research  is  needed  to  further  develop  the  model 
and  to  inerease  our  understanding  of  CT. 

Validation  of  the  CT  Model  as  Applied  to  Army  Battle  Command 

The  results  of  the  experimental  validation  of  the  CT  model  supported  some  of  the  model’s 
predictions.  However,  eonfirmation  of  predieted  relationships  generated  by  a  model  is  only  one 
aspect  of  its  validation.  The  degree  to  whieh  any  psychological  model  can  be  applied  to  real- 
world  situations  is  also  an  important  measure  of  its  validity.  Thus,  a  second  investigation  was 
eonducted  to  determine  whether  the  model  of  CT  was  applieable  to  the  domain  of  Army  battle 
eommand.  The  domain  of  Army  battle  command  was  chosen  because  it  demands  high  levels  of 
CT  ability  for  reasons  previously  discussed.  This  investigation  is  deseribed  in  more  detail  in 
Fiseher,  et  al,  (2008a). 


Method 

The  purpose  of  the  investigation  was  to  determine  whether  the  model  adequately  defined 
CT  skills  (CTS),  situational  conditions,  and  predisposing  factors  important  and/or  problematic  to 
Army  battle  command.  Eighteen  Army  officers  stationed  at  a  large  Army  installation  partieipated 
in  the  research  in  April,  2000.  Participants  completed  a  survey  that  assessed  their  opinions  and 
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experiences  concerning  CIS,  predisposing  attitudes  and  situational  conditions  as  applied  to  the 
domain  of  battle  command.  The  survey  also  prepared  participants  for  guided  interviews  that 
further  elaborated  the  answers  they  had  provided  on  the  survey.  The  session,  which  lasted 
between  1 .5  and  2  hours,  concluded  with  individual  focused  interviews  on  a  wide  range  of  topics 
pertinent  to  CT  in  battle  command. 

The  investigation  also  identified  a  set  of  CTS  that  are  both  important  for  battle  command 
and  whose  execution  is  problematic  or  difficult.  Thirteen  broad  classes  of  CTS  had  been 
identified  from  the  literature.  These  skills  were  then  narrowed  down  by  jointly  considering  two 
criteria  based  on  the  survey  data,  namely  importance  and  difficulty.  First,  participants  rated  the 
importance  for  battle  command  of  the  each  of  the  13  classes  of  skills.  Second,  the  number  of 
officers  who  reported  that  they  had  observed  problems  in  executing  each  of  the  13  classes  of 
skills  was  tabulated.  In  this  way  a  set  of  eight  CTS  were  identified  that  participants  ranked  as 
both  problematic  and  important  to  battle  command  (see  Table  1).. 


Table  1.  Critical  Thinking  Skills  (CTS)  -  Important  and  Problematic  in  Battle  Command 


c 

T 

S 

# 

CTS  Title 

CTS  Definition 

Primary  Battle 
Command  Task 

Battle  Command  Errors  and 
Deficiencies 

1 

Frame  The 
Message 

Ability  to  identify  essential 
elements  of  messages, 
understand  their 
relationships,  and  deseribe 
high  fidelity  representation 
of  the  message. 

Clarify  intent  of  the 
eommanders  1  and  2 
levels  up 

Difficulty  in  establishing  clear  and 
accurate  understanding  of  CDR 
intent 

Difficulty  in  conveying  clear  CDR 
intent 

2 

Reeognize  Gist 

In  Material 

Ability  to  sort  through 
details  in  a  message 
(written,  graphieal,  visual, 
auditory,  and/or  tabular)  and 
extraet  the  gist  therein. 

Restate  mission 
objeetives  provided  by 
upper  eehelon  to  write 
own  mission  statement 

Too  much  detail  in  OPORDs  that 
must  be  filtered  to  establish  gist 
that  supports  writing  of  own 
mission  statement 

Too  little  time  at  lower  echelons 
to  accurately  extract  essence  of 
mission 

3 

Develop  An 
Explanation 

That  Ties 
Information 
Elements 
Together  In  A 
Plausible  Way 

Ability  to: 

•  Arrange  evidenee  logieally 

•  Highlight  gaps  in 
knowledge. 

•  Develop  an  explanation  or 
multiple  explanations 
based  on  evidenee 

•  Evaluate  explanation(s)  for 
plausibility 

Interpret  reports  of 
reeent  enemy  aetivities 
in  area  of  interest  to 
estimate  enemy  intent 
and  prediet  enemy 
aetions 

Overlook  seemingly  unrelated 
facts 

Fail  to  assess  the  quality  of 
information 

Difficulty  in  filtering  excessive 
information 

Tendency  to  embellish  enemy 
activity  reports — over-reports  of 
enemy  contact  and  movement 
Tendency  to  discount  initial 
reports 

4 

Generalize 

From  Speeifie 
Instanees  to 
Broader  Classes 

The  ability  to  reeognize  and 
then  elassify  speeifie 
faets/ineidents/events  as  part 
of  a  general  eategory. 

Interpret  reports  of 
enemy  disposition 

Fail  to  accurately  induce  patterns 
of  overall  movement  based  on 
report  instances 

Tendency  to  disregard  reports  that 
do  not  match  expectations 

Tendency  to  inflate  information  in 
reports 
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Table  1  (Continued),  Critical  Thinking  Skills  (CTS)  Important  and  Problematic  in  Battle 
Command. 


CTS 

# 

CTS  Title 

CTS  Definition 

Primary  Battle 
Command  Task 

Battle  Command  Errors  and 
Deficiencies 

5 

Use  Mental 
Imagery  To 
Evaluate  Plans 

The  ability  to  aeeurately 
ereate  mental  images  in 
one’s  mind  how  resourees 
will  be  applied  and  events 
will  unfold  within  a 
situation. 

Develop  seheme  of 
maneuver 

War  game  eourses  of 
aetion  (CO A) 

Failure  to  visualize  events 

Fail  to  inelude  suffieient  detail  in 
Courses  of  Aetion  (COAs) 

Failure  to  eonsider  eontingeneies 
Fail  to  eonsider  how  plans  eould 
go  wrong 

Generate  only  one  COA 

Failure  to  eonsider  eombat 
multipliers 

Diffieulty  in  keeping  traek  of 
mobile  forees 

6 

Challenge 

One’s  Bias 

Ability  to  eonsistently 
reevaluate  one’s  eurrent 
view  of  situations  for 
prejudiee  or  bias  as  new 
information  is  reeeived. 

Change  own-unit 
plans  based  on  new 
taetieal  input 

Tendeney  to  “fight  the  plan” 
General  reluetanee  to  ehange  plans 

7 

Examine  Other 

Peoples’ 

Perspeetives 

Ability  to  view  and  inter¬ 
pret  eireumstanees  from 
perspeetives  of  different 
individuals,  eultures, 
religions,  and  timeframes. 

Interpret  reports  of 
reeent  enemy 
aetivities  in  area  of 
interest 

Failure  to  aeeurately  estimate 
enemy  intent 

8 

Deeide  when  to 
seek  infor¬ 
mation  based  on 
its  value  and 
eost 

Ability  to  evaluate  need  for 
new  information  in  terms  of 
its  eost  in  time,  resourees 
and  risk 

Assess  eurrent 
situation 

Tendeney  to  spend  too  mueh  time 
planning  and  gathering 
information 

Tendeney  to  make  quiek  deei- 
sions  without  gathering  more 
information 

Results 

The  results  of  this  researeh  indieate  that  the  CT  model  largely  captures  the  skills, 
situational  conditions,  and  predisposing  factors  significant  to  Army  battle  command.  All  of  the 
instances  of  these  three  model  components  were  regarded  as,  at  least,  sometimes  important  to 
battle  command.  However,  some  components  were  less  important  than  others,  according  to  our 
respondents.  Application  of  the  criteria  of  importance  and  being  problematic  in  battle  command 
resulted  in  the  identification  of  eight  high-payoff  CTS  (see  Table  1). 


Integrating  the  CT  Model  and  Skills  into  the  Army  Curriculum 

During  the  early  stages  of  this  project,  the  Army  began  a  transformation  of  education  and 
training  at  all  levels;  that  transformation  continues  today.  One  example  of  this  transformation  has 
been  ongoing  at  the  Command  and  General  Staff  College  (CGSC)  at  Ft  Leavenworth,  Kansas. 
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CGSC  transformation  goals  included  new  curriculum  and  new  methods  of  instruction  to  make 
the  instruetion  more  portable  and  modularized  to  meet  the  speeial  needs  of  offieers  in  different 
speeialties.  One  strong  influenee  on  the  proeess  was  a  poliey  ehange  whereby  100%  rather  than 
50%  of  a  given  year-group  of  Army  officers  would  be  eligible  for  attendanee  at  CGSC  or 
equivalent  instruction.  This  new  system  of  edueation  is  referred  to  as  Intermediate  Level 
Edueation  (ILE). 

In  2001  the  researeh  program  reported  in  this  report  was  still  in  its  early  stages.  However, 
at  that  time  the  preliminary  CT  model  had  been  developed,  the  CT  skills  important  for  Army 
offieers  to  develop  had  been  identified,  and  methods  that  teaehers  eould  use  to  develop  those  CT 
skills  in  their  students  had  been  formulated.  In  a  eollaboration  between  ARI  and  the  ILE 
curriculum  development  team,  the  CT  model,  identified  CT  skills,  and  instructional  methods 
from  this  researeh  program  were  ineorporated  into  the  new  lEE  eurrieulum.  The  original  skills 
identified  in  the  first  phase  of  the  present  researeh  were  ineorporated  into  eaeh  of  the  5  major 
bloeks  of  instruction  in  the  Common  Core  eourse,  i.e.  Eoundations,  Eeadership,  Strategic 
Studies,  Operational  Studies,  and  Tactical  Studies.  Within  eaeh  bloek  is  a  series  of  smaller 
modules.  The  CT  skills  were  integrated  into  16  modules  in  the  form  of  lessons  plans  and/or 
assessment.  Eaeh  module  is  divided  into  a  set  of  lesson  plans.  The  pilot  Common  Core  eourse 
ineorporated  the  skills  into  63  lesson  plans.  Also,  training  eoneepts  discussed  in  Eiseher,  et  al. 
(2008b)  were  adopted  to  teach  the  skills. 

Lessons  Learned  in  Training  Critical  Thinking 

The  primary  goal  for  (CT)^  was  to  provide  effective  training  to  develop  high  levels  of  CT 
skills.  A  seeondary  goal  for  (CT)  was  to  provide  a  stand-alone  system  that  Army  offieers  eould 
aeeess  anywhere  in  the  world.  In  summary,  this  researeh  sought  to  ereate  an  easily  aeeessed, 
state-of-the-art  distance-learning  instruction  that  would  effectively  develop  CT  ability  required  to 
perform  important  tasks  for  battle  eommand.  To  meet  this  objeetive,  it  was  important  to  employ 
the  best-known  teehniques  for  promoting  CT  and  distanee-based  training. 

Two  investigative  strategies  were  adopted  to  determine  the  best  methods.  Eirst,  lessons 
learned  from  the  researeh  literature  on  training  CT  and  distanee-based  learning  were  examined. 
The  edueation  and  psychology  literatures  were  reviewed  to  determine  the  types  of  CT  skills  for 
whieh  training  has  been  developed,  the  methods  used  to  teaeh  CT,  and  the  methods  that  have 
been  shown  to  be  most  effective.  Second,  current  programs  to  teaeh  CT  to  Army  officers  were 
reviewed.  Army  instruetors  and  instruetional  developers  who  are  involved  in  teaehing  CT  were 
surveyed  to  identify  suceesses  and  problems  they  have  eneountered. 

The  results  from  the  two  researeh  strategies  were  then  used  to  establish  the  funetional 
requirements  for  Web-based  CT  instruetional  material. 

Lessons  Learned  from  Past  Research 

A  literature  review  of  eritieal  thinking  training  was  eonducted  using  sources  from 
edueation  and  psyehology.  The  review  addressed  questions  sueh  as:  What  methods  have  been 
used  and  evaluated  for  teaching  CT?  What  kinds  of  CT  skills  have  edueators  and  researehers 
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targeted?  Which  methods  are  most  effective  for  promoting  CT?  How  can  the  best  teaching 
methods  be  implemented  in  a  distance  learning  application?  The  conclusions  of  the  literature 
review  are  summarized  below  and  a  detailed  treatment  of  the  review  is  contained  in  Volume 
Three  of  this  series  (Fischer,  et  ah,  2008b). 

The  research  reviews  suggest  that  CT  can  be  taught  in  the  classroom  or  as  part  of  a 
distance-learning  program.  Distance  learning  has  been  effectively  used  to  promote  learning  of 
complex  cognitive  skills.  Some  studies  show  that  it  is  superior  to  other  forms  of  training.  This 
suggests  that  it  might  be  used  to  improve  CT  in  adult  and  juvenile  populations.  However, 
distance  learning  does  produce  lower  motivational  levels  among  students  who  use  it  (Edlred, 
1994).  Therefore,  any  attempt  to  teach  CT  using  distance-learning  methods  should  provide 
motivational  components  to  counteract  the  isolation  inherent  in  distance  learning.  Multimedia 
and  interactivity  should  also  be  used  to  increase  motivation  and  to  increase  the  efficiency  by 
which  information  is  transmitted  to  the  student.  If  an  adequate  level  of  interactivity  can  be 
provided,  as  determined  by  the  instructional  goals  of  the  training,  Web-based  training  (WBT) 
should  be  used  to  teach  CT.  WBT  facilitates  the  use  of  e-mail,  which  can  provide  the  student 
with  a  social  network  needed  to  maintain  interest  and  combat  isolation.  WBT  is  also  easy  to 
modify  and  distribute  in  a  timely  fashion. 

Recent  instructional  theories  indicate  that  CT  training  should  include  rich  and  varied 
examples,  presented  in  real  world  contexts.  If  the  training  program  is  computer-based,  it  should 
be  highly  interactive  and  multimedia.  Instruction  should  be  context-rich,  problem-based,  and 
goal-oriented  and  should  include  interactive  simulations  for  demonstration,  exploration,  and 
discovery,  and  collaborative  environments  for  practice  and  transfer  of  knowledge  to  new 
situations  (Mayer,  1997). 

The  CT  tasks  targeted  by  the  training  should  be  selected  from  the  environment  to  which 
they  will  be  applied.  Therefore,  training  should  target  those  cognitive  tasks  that  are  important 
and  problematic  to  the  Army. 

Lessons  Learned  from  Army  Instructors  and  Instructional  Developers 

The  Army  currently  incorporates  formal  CT  training  in  their  schoolhouse  curriculum. 
Although  neither  as  systematic  nor  universal,  CT  education  also  occurs  in  field  units  through 
experiences  and  mentoring  (Riedel,  et  ah,  2001).  Examination  of  current  methods  of  teaching  CT 
is  important  in  order  to  capitalize  on  lessons  learned.  A  survey,  described  below,  was  conducted 
to  assess  instructors’  experiences  in  the  teaching  of  CT. 

Method 

Eight  instructors  and  instructional  developers  participated  in  the  survey.  Respondents 
were  Army  officers  or  civilian  researchers  working  at  CGSC,  Kansas  State  University,  and  the 
Army  War  College.  The  CGSC  instructors  taught  at  the  School  for  Command  Preparation  (SCP), 
the  Command  General  Staff  Officers  Course  (CGSOC),  and  the  Combined  Arms  Services  Staff 
School  (CAS  ).  The  respondent  from  Kansas  State  University  was  a  Professor  of  Military 
Science. 
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Respondents  eompleted  a  three-seetion  survey  that  assessed  demographic  information 
and  their  work  experience  teaching  or  developing  curricula  for  CT,  the  classes  they  taught  that 
involved  CT,  and  their  opinions  and  experiences  in  teaching  CT.  When  they  had  completed  the 
survey,  they  returned  their  written  responses  and  contacted  the  researchers. 

Results  and  Conclusions 

The  results  of  the  survey  showed  that  instructors  and  curriculum  developers  were  mostly 
satisfied  with  how  CT  was  taught  at  Army  educational  institutions  at  that  time.  They  believed 
that  it  was  adequately  covered  and  that  the  guiding  conception  of  CT  that  they  used  (based  on 
Paul  and  Elder,  2001)  was  useful  and  sufficient.  Their  experience  suggested  that  realistic, 
practical  exercises  that  require  practicing  CT  skills  are  the  best  way  to  improve  those  skills. 
However,  they  also  recognized  that  feedback  about  students’  performance  is  a  necessary  feature 
of  practical  exercises.  They  also  favored  rich  and  varied  examples  and  realistic  context  for  the 
exercises.  Their  central  complaints  as  teachers  and  curriculum  developers  were  that  the  Army 
culture  tends  to  (1)  create  suspicion  about  the  type  of  slow  deliberate  thought  that  characterizes 
CT,  and  (2)  discourage  innovative  thinking.  For  this  reason,  some  instructors  and  curriculum 
developers  avoided  talking  about  CT  in  the  abstract.  They  avoided,  for  example,  instruction  that 
refers  to  various  models  of  CT,  and  even  avoided  the  term,  CT.  Instead,  they  attempted  to 
encourage  development  of  CT  using  covert  methods.  While  other  respondents  thought  it  was 
useful  to  present  and  discuss  models  of  CT,  all  respondents  agreed  that  convincing  students  of 
the  need  and  importance  of  increasing  CT  skills  was  a  challenge. 

The  literature  on  CT  provides  a  different  view  of  how  CT  should  be  taught  than  the 
recommendations  from  instructors.  These  two  vantage  points  do  not  necessarily  yield 
contradictory  advice  for  new  CT  learning  programs.  Instead,  they  seem  to  yield  different  kinds  of 
recommendations.  Researchers  and  instructional  experts  agree  that  practical  exercises  are  the 
best  way  to  teach  CT  and  acknowledge  the  possibility  that  distance  based  methods  may  be  more 
effective.  They  also  agree  that  examples  that  are  realistic,  practical,  and  context-rich  are 
effective.  However,  the  literature  has  more  to  say  about  methods  of  training.  While  instructional 
experts  report  satisfaction  with  current  CT  theories  (Paul  &  Elder,  2001)  and  the  use  of  practical 
exercises  as  a  useful  method,  the  literature  suggests  that  a  host  of  other  techniques  may  be  more 
effective.  These  techniques  are  discussed  further  in  Volume  Three  of  this  series. 

The  results  of  the  literature  review  and  instructor  survey  informed  an  effort  to  establish 
functional  requirements  for  a  distance-learning  program  of  CT  training,  (CT)^. 

Functional  Requirements  For  web-based  training  of  critical  thinking 

To  lay  the  groundwork  for  designing  an  Internet-based  CT  training  program,  an 
Instructional  Systems  Design  (ISD)  analysis  was  performed.  The  first  steps  in  ISD  include 
specification  of  an  initial  estimate  of  the  capabilities  of  proposed  instruction,  the  scope  of  the 
instructional  problem,  the  location  of  instruction,  anticipated  student  load,  alternative 
instructional  strategies,  support  for  maintaining  the  system,  facilities  requirements,  evaluation 
assessment,  quality  improvement  metrics,  and  funding  requirements. 
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Specification  of  these  issues  was  partially  addressed  by  conducting  interviews  with 
members  of  the  ILE  team  who  were  responsible  for  revising  CGSOC  curriculum  and  by 
observing  lessons  in  CT  given  at  the  Tactical  Commanders  Development  Course  of  the  SCP  at 
CGSC.  A  summary  of  the  results  of  the  ISD  analysis  on  each  of  these  topics  is  given  below.  The 
functional  requirements  are  discussed  in  more  detail  in  Fischer,  et  ah,  (2008b). 

Estimate  of  Instructional  Capability 

Interview  respondents  suggested  courseware  is  needed  that  (1)  can  be  accessed  by  any 
officer  anywhere  in  the  world  and  (2)  contains  lessons  not  addressed  by  current  training.  The 
new  CT  training  should  be  easily  accessed  on  the  Internet  to  provide  maximal  distribution  to 
Army  personnel.  It  should  complement  and  support  the  Army’s  doctrine  for  leadership,  e.g.  FM 
6-22  (DA,  2006)  and  currently  provided  Army  educational  experiences  in  CT.  Finally,  it  should 
address  deficits  in  officers’  CT  as  evidenced  in  observed  deficiencies  and  errors  in  human 
performance  associated  with  common  battle  command  tasks. 

Scope  of  Instructional  Problem 

The  instruction  should  focus  on  a  set  of  critical  thinking  skills  that  are  important  and 
problematic  to  battle  command  and  the  Military  Decision  Making  Process  (MDMP).  Fach  skill 
should  be  trained  separately  and  independently  from  the  rest  to  meet  the  practical  needs  of 
instructors  and  students. 

Location(s)  of  Instruction 

The  bulk  of  instruction  should  be  located  on  the  Internet,  accessible  through  an  Army- 
provided  URF.  Some  instruction  may  reside  in  links  to  other  sites  (e.g.,  references,  related  on¬ 
line  courses). 

Target  Audience 

The  student  target  audience  should  be  any  officer  currently  enrolled  in  one  of  the  Army’s 
training  and/or  educational  facilities. 

Alternative  Instructional  Strategies,  Methods,  and  Media 

Alternative  instructional  strategies,  methods,  and  media  should  be  addressed  if  expected 
training  effectiveness  is  found  to  be  lacking  during  the  formative  evaluations.  The  instructional 
strategy  should  be  designed  to  optimize  student  participation,  student  feedback,  student  pacing, 
and  instructional  sequence. 

SMEs  for  Training  Development 

A  number  of  resources  are  required  to  develop  the  training  system  including  Army 
subject  matter  expert  (SMF)  participation  to  oversee/validate  content  development.  To  evaluate 


14 


the  system’s  effeetiveness,  eomments  from  students  and/or  instruetors  who  partieipate  in  pilot 
tests  of  the  system  should  be  obtained. 


Evaluation  and  Assessment  Measures 

Evaluation  and  assessment  measures  should  be  based  on  development  of  a  measurement 
paradigm,  whieh  should  also  give  rise  to  a  set  of  robust  measures  of  CT.  The  experienee  of  Army 
SMEs/instructors  should  be  utilized  for  validation  of  the  measures. 

Quality  Improvement  Metrics 

The  measures  cited  above  should  be  adapted  to  assess  how  the  process  of  CT  has 
improved  among  the  officers  taking  the  course.  The  assessment  measures  should  be  focused  not 
global.  They  should  be  designed  to  assess  the  particular  aspects  of  CT  that  have  been  aided  or 
not  aided  by  the  training,  rather  than  just  CT  as  a  whole. 

Additional  Requirements 

The  interviews  with  the  lEE  team  members  and  classroom  observations  also  yielded 
additional  requirements  for  new  distance-based  training  programs  in  CT.  Both  sets  of  data 
suggest  that  special  consideration  be  given  to  the  population  of  adult  learners  for  which  training 
would  be  developed.  Interview  respondents  favored  a  particular  model  of  curriculum  for  adult 
learners,  the  Experiential  Learning  Model  (ELM)  (Kolb,  1984)  which  has  been  adopted  by 
CGSC  and  others.  Hence,  two  additional  requirements  for  new  CT  training  are  that  (1)  factors 
that  affect  adult  learners  should  be  considered  in  the  system’s  design,  and  (2)  the  ELM  should 
serve  to  guide  new  curriculum  development.  Eactors  that  affect  adult  learning  are  discussed  first, 
followed  by  a  summary  of  ELM. 

Computerized  Training  of  Critical  Thinking  (CT)^ 

Based  on  the  functional  requirements  noted  above,  a  prototype  training  system  was 
designed  and  developed  to  increase  the  CT  skills  of  Army  officers.  The  training  system. 
Computerized  Training  of  Critical  Thinking  (CT)  delivers  distance-based  training  via  the 
Internet  for  two  CTS. 

The  design  of  (CT)^  was  guided  by  the  results  of  a  series  of  studies,  reported  in  this 
document  and  elsewhere  (Fischer,  et  ah,  2008a;  Fischer,  et  ah,  2008b).  The  studies  led  to  the 
identification  of  a  set  of  pedagogical  principles  that  served  as  the  philosophical  basis  of  (CT)  ’s 
design.  These  17  principles  are  bulleted  below. 

♦  CT  Skills  Can  Be  Learned,  Trained,  and  Transferred. 

♦  Practice  is  Essential. 

♦  Feedback  is  Essential. 

♦  Assessment  is  Essential. 

♦  Training  Conditions  Should  Optimize  Transfer. 

♦  Part-Training  Methods  are  Most  Effective. 
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♦  Focus  on  Important  and  Problematic  Cognitive  Tasks. 

♦  Focus  on  Common  and  Consequential  Errors. 

♦  Use  Training  Methods  Appropriate  for  Adults  (Army  Officers). 

♦  Use  Concrete  Experience  to  Start. 

♦  Training  Should  be  Scenario  Based. 

♦  Training  is  increasingly  Complex  and  Dijficult. 

♦  Training  is  Distance-based. 

♦  Multimedia  is  Essential. 

♦  Interactivity  is  Essential. 

♦  Scoring  Increases  Motivation. 

♦  Use  innovative  Training  Techniques. 

Course  Objectives 

The  focus  of  (CT)  is  to  improve  key  skills  that  support  critical  thinking  and  thus  help 
Army  personnel  process  information  more  effectively  and  efficiently.  CT  involves  a  deliberate, 
systematic  awareness  of  the  process  and  products  of  one’s  own  thinking.  The  training  program 
focuses  on  targeting  common  -  and  potentially  serious  -  errors  that  people  make  when  they  fail 
to  apply  appropriate  critical  thinking  skills.  These  errors  include  overlooking  important  details, 
misinterpreting  information,  and  making  incorrect  assumptions  -  all  of  which  can  lead  to  poor 
decision-making.  The  training  program  highlights  awareness  of  these  errors  and  teaches  specific 
techniques  that  can  help  people  overcome  them.  It  presents  the  student  with  real-world  situations 
and  asks  them  to  complete  numerous  thinking  exercises  that  require  the  practice  and  application 
of  CT  skills  in  a  variety  of  realistic  settings. 


The  Target  Student  Population 

(CT)  was  designed  for  Army  personnel  in  leadership  positions,  although  all  personnel 
could  benefit  from  the  training.  In  order  to  make  the  training  as  meaningful  as  possible,  realistic 
Army  messages  and  situations  were  employed;  therefore,  before  beginning  the  training  program, 
students  should  have  some  Army  training  and  experience.  They  should  be  familiar  with  Army 
terminology,  acronyms,  and  standard  report  formats.  They  should,  for  example,  be  able  to 
understand  standard  Army  messages  such  as  task  organization  tables,  operation  orders,  mission 
statements,  and  battlefield  sketches.  (CT)^  assumes  students  already  have  some  domain 
knowledge;  it  gives  them  practice  in  applying  strategies  to  use  that  knowledge  most  effectively. 

Overview  of  (CT)^  Format 

The  course  is  presented  on  the  Internet  in  a  Web-based  format.  It  is  divided  into  three 
sections:  an  introduction  to  the  general  topic  of  CT,  followed  by  two  main  modules  each  of 
which  focuses  on  a  particular  CTS.  The  Introduction  to  CT  highlights  the  importance  of 
developing  CT  skills  and  briefly  outlines  the  organization  and  methods  used  in  the  course.  The 
first  module  focuses  on  the  CTS  of  framing  a  message,  and  the  second  module  focuses  on 
distilling  the  main  gist  of  messages. 
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Each  module  comprises  a  series  of  elements  or  lessons  designed  to  help  students  develop 
speeifie  teehniques  that  will  help  them  interpret  and  eritieally  perform  the  CTS  in  question  more 
effeetively.  Speeifieally,  eaeh  element  provides  training  that  is  designed  to  reduee  one  or  more 
eommon  errors  assoeiated  with  the  CTS. 

Module  Format 

Each  module  is  self-eontained  and  presents  training  pertinent  to  a  partieular  eritical 
thinking  skill.  Although  it  is  strongly  reeommended  that  the  students  progress  through  the 
modules  in  a  suggested  sequenee  deseribed  in  the  training,  the  instruetor  ean  assign  students  to 
speeifie  modules  or  to  modules  in  a  different  sequenee  than  is  reeommended. 

Organization  and  Content  of  Modules 

Eaeh  module  begins  with  an  introduction  to  the  partieular  CTS  targeted  by  the  module. 
The  introduction  section  of  the  module  opens  with  a  eonerete  scenario  illustrating  the  importanee 
of  the  skill  and  the  errors  that  ean  result  when  the  skill  is  not  applied.  It  then  goes  on  to  provide  a 
elear  definition  of  the  CTS,  as  well  as  key  eoneepts  and  terms  and  a  brief  explanation  of  how  this 
cognitive  task  relates  to  eritical  thinking.  This  is  followed  by  eonerete  examples  illustrating  the 
key  concepts.  The  introduetion  seetion  eoneludes  with  an  outline  of  the  main  objeetives  of  the 
module  and  a  plan  for  how  the  objeetives  will  be  trained.  The  next  seetion  of  eaeh  module 
eonsists  of  two  to  five  sub-modules  ealled  Cognitive  Elements.  Eaeh  element  targets  a  eommon 
CT  error  associated  with  the  CTS. 

Element  Format 

Eaeh  of  several  elements  within  eaeh  module  targets  a  speeifie  CT  error  and  foeuses  on 
teehniques  that  one  ean  use  to  develop  awareness  of  where,  when,  how,  and  why  CT  skills 
should  be  applied.  Eaeh  element  eontains  the  following  seetions: 

1 .  An  opening  seenario  that  illustrates  the  importanee  and  prevalence  of  the  CT  error 
eovered  in  the  element 

2.  Main  purpose  of  the  element 

3.  Speeifie  lesson  objeetives 

4.  Instruetion  providing  eontent  information,  rationale,  and  praetieal  applications 

5.  Training  examples 

6.  Praetiee  exereises 

7.  Summary  of  main  point  of  the  lesson,  praetieal  eonsiderations  and  applieations 

8.  Exit  tests 

Assessment  Format 

The  exit  tests  at  the  end  of  eaeh  element  provide  a  reinforeement  of  the  material  and 
inform  the  student  and  the  instruetor  about  the  extent  to  whieh  the  student  aehieved  the  element 
lesson  objeetives.  They  inelude  a  knowledge,  knowledge  applieation,  and  skill  applieation  test. 
All  tests  are  designed  using  best  praetiees  test  eonstruction  and  are  graded  by  the  eomputer  -  that 
is,  they  do  not  require  the  instruetor  to  do  the  grading. 
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Access  to  Course 


Students  are  assigned  a  password  from  the  course  administrator.  Only  students  who  have 
a  password  are  able  to  log  on  to  the  program. 

Course  Syllabus 

Course  Introduction  The  course  introduction  provides  a  brief  overview  of  CT, 
explaining  what  it  is  and  why  it  is  important,  and  outlining  how  the  course  is  organized. 

Skill  1:  Frame  the  Message  The  first  module  provides  instruction  on  the  CTS,  Frame 
the  Message.  It  is  designed  to  promote  the  ability  to  identify  the  essential  elements  of  a  message, 
understand  the  relationships  among  them,  and  describe  a  high-fidelity  representation  of  the 
message.  It  addresses  common  battle  command  problems  such  as  difficulty  in  establishing  clear 
and  accurate  understanding  of  a  commander’s  intent  statement  and  difficulty  in  conveying  clear 
intent.  The  main  learning  objectives  of  this  module  are:  (1)  developing  awareness  of  message 
structure  in  order  to  better  interpret,  evaluate,  critique,  and  remember  key  information;  and  (2) 
recognizing  potential  biases  in  interpreting  information. 

Skill  2:  Recognize  Gist  in  Material  The  second  module  provides  instruction  in  the  CTS, 
Recognize  Gist  in  Material.  It  is  designed  to  promote  the  ability  to  sort  through  the  details  in  a 
message  (written,  graphical,  visual,  auditory,  and/or  tabular)  and  extract  the  gist  therein.  It 
addresses  common  battle  command  problems  such  as  filtering  overly-detailed  operation  orders  in 
order  to  write  clear  and  concise  mission  statements  that  capture  the  gist  of  the  original  message, 
and  reducing  the  amount  of  time  that  time -pressed  units  have  to  spend  on  accurately  extracting 
the  essence  of  a  mission.  The  main  learning  objectives  of  this  module  are:  representing  the 
essential  points  of  a  message  in  a  clear,  concise  manner;  and  recognizing  how  the  process  of 
distilling  the  main  idea  encourages  deeper  processing  and  can  lead  to  increased  ability  to 
remember,  evaluate,  and  use  the  information. 

Recommended  Sequence  of  Training  It  is  strongly  recommended  that  students  progress 
through  the  training  in  sequence,  beginning  with  the  introduction,  moving  on  to  CTS  1,  CTS  2, 
and  so  on.  However,  the  course  has  been  designed  so  that  each  CTS  can  function  as  a  stand¬ 
alone  module. 

Instructor’s  Role  The  course  is  designed  to  function  as  a  stand-alone  program  and  can 
be  completed  with  minimal  instructor  involvement;  however,  instructors  can  augment  the 
material  by  facilitating  discussions,  providing  additional  real-world  examples,  and  by  providing 
additional  activities  to  reinforce  and/or  supplement  the  material  covered  in  the  course. 

Suggested  Implementation  The  course  can  be  used  in  a  classroom  setting,  as 
supplemental  exercises  completed  outside  of  class,  or  it  can  be  used  as  a  distance-learning 
course. 
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Estimated  Time  to  Complete  the  Course  Each  element  is  designed  to  take 
approximately  1 .5  to  2  hours  to  work  through  the  eontent  information  and  do  the  praetiee 
training  exereises. 


Evaluation  of  (CT)^ 

An  investigation  was  eondueted  to  evaluate  the  effeetiveness,  usability,  and  student 
aeeeptanee  of  (CT)  .  The  eentral  objeetive  of  the  researeh  was  to  determine  whether  the  new  CT 
training  system  effeetively  inereases  measurable  indieators  of  CT  eompared  to  two  other  learning 
eonditions.  The  researeh  also  assessed  partieipating  students’  attitudes  and  subjective  evaluations 
of  (CT)^  as  indieators  of  aeeeptanee  and  usability.  The  evaluation  is  deseribed  in  greater  detail  in 
Fiseher,  et  ah,  2008b). 


Method 


Participants 

Nineteen  Soldiers  (17  males  and  2  females)  from  the  85th  Reserves  Training  Division 
volunteered  to  partieipate  in  the  researeh. 

Materials 

Participants  were  randomly  assigned  to  one  of  three  CT  training  eonditions.  They  either 
received  Web-based  training  in  one  CT  module  of  (CT)  ,  completed  distance-based  coursework 
modeled  after  an  existing  CT  class  offered  at  CGSC,  or  received  no  training  beyond  what  they 
had  already  been  given  as  Reserve  Army  officers.  All  participants  were  first  asked  to  complete  a 
demographic  questionnaire  that  gathered  information  about  their  rank,  job  position,  and  gender. 
Different  sets  of  instructional  materials  were  also  developed  and  delivered  to  participants 
depending  on  the  condition  to  which  they  were  assigned. 

Results  and  Conclusions 

The  results  of  this  investigation  indicate  that  military  students  find  (CT)  highly 
acceptable.  Although  the  sample  of  participants  who  used  (CT)^  was  small,  it  was  uniformly 
positively  rated.  Despite  the  extensive  time  commitment  the  training  program  requires,  users 
found  it  interesting  and  well  worth  their  time.  Participants  thought  the  program  offered  training 
not  available  elsewhere  in  the  Army.  The  self-paced  feature  of  the  program  appears  to  be  one  of 
the  reasons  it  received  favorable  reviews.  Participants  thought  it  was  highly  relevant  and 
beneficial  to  their  military  and  civilian  work. 

Although  (CT)  was  well  regarded,  it  is  clear  from  reviewers’  recommendations  that  the 
usability  of  the  program  needs  improvement.  Specifically,  the  program  needs  to  have  a  better 
capability  to  navigate  to  previous  pages  and  more  flexibility  in  the  use  of  its  audio  components. 
Users  also  wanted  the  system  to  allow  them  to  make  errors  which  it  currently  does  not. 
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(CT)  also  appears  to  be  generally  effeetive  at  eneouraging  eritical  thinking,  at  least  about 
messages  Army  personnel  must  evaluate.  The  researeh  elearly  showed  that  the  WBT  enhaneed 
memory  for  messages,  possibly  beeause  it  eneourages  greater  depth  of  proeessing.  The  finding 
that  memory  of  messages  is  enhaneed  after  participating  in  (CT)^  is  a  strong  indicator  that  those 
messages  have  been  processed  at  a  deeper  level. 

(CT)  also  seems  to  inhibit  the  production  of  inferences  that  go  well  beyond  what  is 
explicitly  given  in  the  message.  Participants  who  took  the  (CT)^  training  made  far  fewer 
unjustified  inferences  than  participants  assigned  to  the  other  two  training  conditions. 
Examinations  of  the  responses  reveals  that  (CT)^  participants  did  make  inferences;  however,  they 
justified  them  by  pointing  out  explicit  information  given  in  the  message  that  supported  their 
inferences.  Therefore,  (CT)^  appears  to  encourage  discrimination  of  what  is  “known”  or  “given” 
from  what  might  be  added  (i.e.,  inferred)  by  the  perceiver.  The  ability  to  make  this  distinction  is 
critical  to  avoiding  many  errors  of  reasoning.  In  these  data,  for  example,  participants  who  were 
inclined  to  make  unjustified  inferences  expressed  beliefs  that  were  clearly  erroneous.  These 
results  suggest  that  (CT)^  encourages  evidence-based  reasoning. 

In  summary,  the  present  research  provided  an  initial  evaluation  of  (CT)  ,  documenting  its 
acceptance  by  military  users  and  its  effectiveness  at  improving  CT.  Future  research  should  focus 
on  comparing  the  effectiveness  of  other  CT  training  programs  to  (CT)^  using  alternative 
indicators  of  CT. 


Summary 

This  report  presented  an  overview  of  a  research  program  to  develop  and  validate  a  web 
based  training  in  critical  thinking  for  Army  officers.  Volumes  Two  and  Three  of  this  series 
describe  this  research  in  more  detail.  Volume  Two  describes  a  model  of  critical  thinking,  results 
of  a  literature  review  that  provided  the  theoretical  basis  for  the  model,  and  experiments  to 
validate  the  model.  Volume  Three  of  this  series  describes  the  training  that  was  developed 
(Fischer,  et  ah,  2008b).  An  expanded  version  of  (CT)^  is  documented  in  Fischer,  et  ah,  2008). 
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